Buggy Tests

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 260 of xUnit Test Patterns for the latest information.

Bugs are regularly found in the automated tests.

Fully Automated Tests (see Goals of Test Automation on page X) are supposed to act as a "safety net" for teams doing iterative development. But how can we be sure the safety net actually works?

Buggy Tests is a project level indication that maybe all is not well with our automated tests.

Symptoms

A build fails and a failed test is to blame. Upon closer inspection, it turns out that the code it was testing works fine but the test indicated it was broken.

We encountered Production Bugs (page X) despite having tests that verify the specific scenario in which the bug was found. Root cause analysis indicates that the test has a bug in it which precluded catching the error in the production code.

Impact

Tests that give misleading results are dangerous! Tests that pass when they shouldn't (a false negative as in "nothing wrong here") give a false sense of security. Tests that fail when they shouldn't (a false positive) discredit the tests. They are like the little boy crying "wolf"; after a few occurrences there is a tendency to ignore them.

Causes

Buggy Tests can be caused by many things. Most of these also show up as code or behavior smells but as project managers we are unlikely to see these underlying smells until we specifically go looking for them.

Cause: Fragile Test

The Buggy Tests may just be the project-level symptom of Fragile Test (page X). For false positive test failures, a good place to start is the "four sensitivities" (Interface Sensitivity (see Fragile Test), Behavior Sensitivity (see Fragile Test), Data Sensitivity (see Fragile Test) and Context Sensitivity (see Fragile Test).) Each of these could be the change that caused the test to fail. Removing the sensitivities through the use of Test Doubles (page X) and refactoring can be challenging but ultimately it will make the tests much more dependable.

Cause: Obscure Test

A common cause of false negative test results (tests that passed when they shouldn't have) is that we have Obscure Tests (page X) which are hard to get right especially when we are modifying existing tests that were broken by a change we made. Since automated tests are hard to test, we don't often verify that a modified test still catches all the bugs it was initially designed to trap. As long as we get a green bar we think we are "good to go" but we may have created a test that never fails.

Obscure Tests are best addressed through refactoring of tests focusing on the reader of the tests. The real goal is Tests as Documentation (see Goals of Test Automation) any anything less than that will increase the likelihood of Buggy Tests.

Cause: Hard to Test Code

Another common cause, especially with "legacy software" (defined for our purposes as any software that doesn't have a complete suite of automated tests) is that the design of the software is not conducive to automated testing. This Hard-to-Test Code (page X) may force us to use Indirect Testing (see Obscure Test) and that may result in a Fragile Test or a Fragile Fixture (see Fragile Test).

The only way Hard-to-Test Code will become easy to test is to refactor it to improve its testability. (This is described in the Test Automation Strategy and Using Test Doubles narratives.) If this is not an option, we may be able to reduce the amount of test code impacted by a change by applying SUT API Encapsulation (see Test Utility Method on page X).

Trouble-Shooting Advice

When we hear that we have Buggy Tests it is important to ask more questions. We must ask the "five why's" [TPS] to get to the bottom of it to determine exactly which code and/or behavior smells are the cause and to find the root cause of each of them.

Solution Patterns

The solution depends very much on why the Buggy Tests occurred. Refer to the underlying behavior and code smells for possible solutions.

As with all "project smells", we should be looking for project-level causes. These include not giving developers enough time to:

learn to write the tests properly
refactor the legacy code to make test automation easier and more robust
write the tests first.

Failure to address these project-level causes will only ensure that the problems will recur in the near future.

Page generated at Wed Feb 09 16:39:51 +1100 2011